03. Data Pre-Processing

PRTDM2-785 AI Trading C2 L1 Vid3 Ingesting Data

Storing and Preprocessing Data for Machine Learning Projects

Data Storage Options:

  • On-device storage: Suitable for small datasets. Ensure enough space is available.
  • External hard drive: Consider this when your laptop runs out of space. Opt for one with fast read speeds.
  • Cloud storage: Useful but may not be as straightforward as external drives.

Organizing Your Data:

  • Create a central folder on your chosen storage medium.
  • Rename files to be descriptive and recognizable to avoid confusion later.

Data Preprocessing with Pandas:

  1. Introduction to Pandas: Utilize this Python package to handle CSV, Excel, web, or SQL data, converting them into data frames.
  2. Avoid Messy Code: Resist the urge to write quick, ad-hoc preprocessing code. It might complicate things when updating data frequently.
  3. Reusable Code: Establish a preprocessing pipeline for consistent, repeatable steps.
  4. Creating Utility Scripts: Develop utility scripts in Python for reusable preprocessing functions and keep them organized.

By following these structured steps, ensure an efficient workflow from data storage to preprocessing.

Which of the following are NOT considered data preprocessing?
SOLUTION: Selecting an appropriate machine learning model.

Which of the following are TRUE statements about data APIs?
SOLUTION:
  • API stands for Application Programming Interface.
  • Some APIs require tokens or keys to use.
  • API tokens or keys should be treated like passwords.

Next Concept